Test execution ordering: regroup by axis/package to reduce fixture churn by stormi · Pull Request #544 · xcp-ng/xcp-ng-tests

stormi · 2026-05-20T16:27:21Z

pytest's default ordering could interleave image_format=vhd/qcow2 tests in a way that repeatedly destroys and recreates SRs. It could also choose to cross package boundaries as an attempt to optimize test runs, but we'd prefer to see tests logically grouped, for easier log analysis and fixture management (which is perfectible and which we'll have to improve anyway).

We start with pytest's computed ordering, then regroup tests by parametrized axis (currently only image_format) and by leaf python package, in order to keep related tests together and reduce needless costly context switching.

Axis-less tests intentionally share the same ordering level as the first parametrized axis so they stay near their original position instead of being grouped at the beginning of the run.

Interfering with pytest's ordering is not something to do lightly, so it's possible that we may discover issues caused by this. But meanwhile, this has shown a notable reduction (10%) of the number of setup/teardown operations when both vhd and qcow2 image formats are set, and test order closer to what we'd intuitively expect.

ydirson

Quick test of the overall impact, by comparing before and after the output of:

uv run pytest --setup-plan --image-format=vhd,qcow2 tests/ | sort | uniq -c

$ wc -l *cnt
  1026 after.cnt
  1304 before.cnt

Diffing shows a few things:

difference in skipped tests

+      1 SKIPPED [1] tests/storage/zfsvol/test_zfsvol_sr.py:77: zfsvol doesn't provide vhd-parent
+      1 tests/storage/zfsvol/test_zfsvol_sr.py::TestZfsvolVm::test_coalesce[None-snapshot] SKIPPED

investigating those skipped tests, we can se many more TestZfsvolVm tests scheduled than originally
fixtures dispatch_nfs['nfs_device_config'] and dispatch_nfs['nfs4_device_config'] were actually not scheduled before the change, but now are
...

Actually, the set of tests scheduled is really different. There seems to be both tests that were not scheduled but now are (which seems good(?), but is scary): zfsvol stuff, all MooseFS tests, some glusterfs tests, TestTLSVerification stuff... and tests which were but are not any more (even more scary), most prominently any qcow2 test

glehmann · 2026-05-21T15:05:08Z

+      Check pytest.ini or pytest --markers for marker descriptions.
+    - Regroup tests by axis (image_format) and by package (leaf directory)
+    """
+    def get_axis(item: pytest.Item) -> Optional[str]:


get_axis could be moved out of pytest_collection_modifyitems, next SCHEDULING_AXES, to keep the pytest_collection_modifyitems shorter

I hesitated, since it's really purely a help function inside the other function, but I don't mind changing.

glehmann · 2026-05-21T15:10:44Z

+    # List the items in the order that pytest initially determined, and add extra grouping criteria.
+    for item in items:
+        axis_order = axis_ordering[get_axis(item)]
+        package = item.getparent(pytest.Package)


I don't think we have tests outside a package, and I don't think we want that to happen, so you could add

assert package is not None

and simplify the grouped type a bit to

dict[int, dict[pytest.Package, list[pytest.Item]]]

glehmann · 2026-05-21T16:04:46Z

@ydirson I see the same tests skipped with that PR and the commit before, and the comparison with the old and new plan is quite similar (except it looks better in many places).

  1750 /tmp/old.cnt
  1750 /tmp/new.cnt

--- /tmp/old.cnt	2026-05-21 17:56:49.893897471 +0200
+++ /tmp/new.cnt	2026-05-21 17:56:56.971822220 +0200
@@ -1,9 +1,9 @@
       2 
-      1 ============================== 6 skipped in 0.51s ==============================
+      1 ============================== 6 skipped in 0.46s ==============================
       1 collected 671 items
       1 configfile: pytest.ini
       1 -------------------------------- live log setup --------------------------------
-      1 May 21 17:54:28.983 INFO skip test_chain because it depends on TestManualChaining::test_create
+      1 May 21 17:54:41.690 INFO skip test_chain because it depends on TestManualChaining::test_create
       1 platform linux -- Python 3.11.11, pytest-9.0.2, pluggy-1.6.0
       1 plugins: dependency-0.6.1
       1 rootdir: /home/glehmann/src/xcp-ng/xcp-ng-tests
@@ -85,45 +85,45 @@
       1     SETUP    M unix_vm_with_vtpm (fixtures used: snapshotted_halted_uefi_unix_vm)
       2     SETUP    M unsealed_windows_vm_and_snapshot (fixtures used: running_windows_vm_without_tools)
       1     SETUP    M vdi_on_cephfs_sr (fixtures used: cephfs_sr)
-      6     SETUP    M vdi_on_ext_sr (fixtures used: ext_sr)
+      4     SETUP    M vdi_on_ext_sr (fixtures used: ext_sr)
       1     SETUP    M vdi_on_glusterfs_sr (fixtures used: glusterfs_sr)
       2     SETUP    M vdi_on_largeblock_sr (fixtures used: largeblock_sr)
       1     SETUP    M vdi_on_linstor_sr (fixtures used: linstor_sr)
       1     SETUP    M vdi_on_moosefs_sr (fixtures used: moosefs_sr)
       2     SETUP    M vdi_on_nfs_sr (fixtures used: nfs_sr)
-      7     SETUP    M vdi_on_xfs_sr (fixtures used: xfs_sr)
-      7     SETUP    M vdi_on_zfs_sr (fixtures used: zfs_sr)
+      4     SETUP    M vdi_on_xfs_sr (fixtures used: xfs_sr)
+      4     SETUP    M vdi_on_zfs_sr (fixtures used: zfs_sr)
       1     SETUP    M vdi_on_zfsvol_sr (fixtures used: zfsvol_sr)
       1     SETUP    M vdis (fixtures used: host, local_sr_on_hostA1)
       6     SETUP    M vm_on_cephfs_sr (fixtures used: cephfs_sr, host, vm_ref)
-     12     SETUP    M vm_on_ext_sr (fixtures used: ext_sr, host, vm_ref)
+      8     SETUP    M vm_on_ext_sr (fixtures used: ext_sr, host, vm_ref)
       6     SETUP    M vm_on_glusterfs_sr (fixtures used: glusterfs_sr, host, vm_ref)
       6     SETUP    M vm_on_largeblock_sr (fixtures used: host, largeblock_sr, vm_ref)
-      5     SETUP    M vm_on_linstor_sr (fixtures used: host, linstor_sr, vm_ref)
-     12     SETUP    M vm_on_lvmohba_sr (fixtures used: host, lvmohba_sr, vm_ref)
-     12     SETUP    M vm_on_lvmoiscsi_sr (fixtures used: host, lvmoiscsi_sr, vm_ref)
-     12     SETUP    M vm_on_lvm_sr (fixtures used: host, lvm_sr, vm_ref)
-      4     SETUP    M vm_on_moosefs_sr (fixtures used: host, moosefs_sr, vm_ref)
+      6     SETUP    M vm_on_linstor_sr (fixtures used: host, linstor_sr, vm_ref)
+      8     SETUP    M vm_on_lvmohba_sr (fixtures used: host, lvmohba_sr, vm_ref)
+      8     SETUP    M vm_on_lvmoiscsi_sr (fixtures used: host, lvmoiscsi_sr, vm_ref)
+      8     SETUP    M vm_on_lvm_sr (fixtures used: host, lvm_sr, vm_ref)
+      6     SETUP    M vm_on_moosefs_sr (fixtures used: host, moosefs_sr, vm_ref)
       2     SETUP    M vm_on_nfs_sr (fixtures used: host, nfs_sr, vm_ref)
-     12     SETUP    M vm_on_xfs_sr (fixtures used: host, vm_ref, xfs_sr)
-     12     SETUP    M vm_on_zfs_sr (fixtures used: host, vm_ref, zfs_sr)
-      3     SETUP    M vm_on_zfsvol_sr (fixtures used: host, vm_ref, zfsvol_sr)
-    118     SETUP    M vm_ref[None]
+      8     SETUP    M vm_on_xfs_sr (fixtures used: host, vm_ref, xfs_sr)
+      8     SETUP    M vm_on_zfs_sr (fixtures used: host, vm_ref, zfs_sr)
+      5     SETUP    M vm_on_zfsvol_sr (fixtures used: host, vm_ref, zfsvol_sr)
+    126     SETUP    M vm_ref[None]
       1     SETUP    M vm_with_vbds (fixtures used: host, imported_vm, vdis)
       4     SETUP    M windows_vm (fixtures used: imported_vm)
-      3   SETUP    P cephfs_device_config
-      3   SETUP    P cephfs_sr (fixtures used: cephfs_device_config, host, pool_with_ceph)
-      9   SETUP    P ext_sr (fixtures used: host, image_format, unused_512B_disks)
+      2   SETUP    P cephfs_device_config
+      2   SETUP    P cephfs_sr (fixtures used: cephfs_device_config, host, pool_with_ceph)
+      2   SETUP    P ext_sr (fixtures used: host, image_format, unused_512B_disks)
       3   SETUP    P formatted_and_mounted_ext4_disk (fixtures used: host, unused_512B_disks)
       1   SETUP    P fsp_config (fixtures used: host_with_fsp)
       1   SETUP    P fsp_sr (fixtures used: fsp_config, host_with_fsp)
-      3   SETUP    P gluster_disk (fixtures used: _glusterfs_config, pool_with_unused_512B_disk, unused_512B_disks)
-      3   SETUP    P _glusterfs_config
-      3   SETUP    P glusterfs_device_config (fixtures used: host)
-      3   SETUP    P glusterfs_sr (fixtures used: _glusterfs_config, gluster_volume_started, glusterfs_device_config, host, pool_with_glusterfs)
-      3   SETUP    P gluster_volume_started (fixtures used: _glusterfs_config, gluster_disk, host, hostA2)
-     21   SETUP    P hostA2_with_xfsprogs (fixtures used: _xfs_config_on_hostA2, hostA2, image_format)
-     22   SETUP    P hostB1_with_xfsprogs (fixtures used: _xfs_config_on_hostB1, hostB1, image_format)
+      2   SETUP    P gluster_disk (fixtures used: _glusterfs_config, pool_with_unused_512B_disk, unused_512B_disks)
+      2   SETUP    P _glusterfs_config
+      2   SETUP    P glusterfs_device_config (fixtures used: host)
+      2   SETUP    P glusterfs_sr (fixtures used: _glusterfs_config, gluster_volume_started, glusterfs_device_config, host, pool_with_glusterfs)
+      2   SETUP    P gluster_volume_started (fixtures used: _glusterfs_config, gluster_disk, host, hostA2)
+      4   SETUP    P hostA2_with_xfsprogs (fixtures used: _xfs_config_on_hostA2, hostA2, image_format)
+      4   SETUP    P hostB1_with_xfsprogs (fixtures used: _xfs_config_on_hostB1, hostB1, image_format)
       1   SETUP    P host_no_sdn_controller (fixtures used: host)
       1   SETUP    P host_with_dom0_tests (fixtures used: host_with_saved_yum_state)
       1   SETUP    P host_with_dynamically_disabled_ept_sp (fixtures used: host)
@@ -133,51 +133,51 @@
       1   SETUP    P host_with_netdata (fixtures used: host_with_saved_yum_state)
       1   SETUP    P host_without_mlx_card (fixtures used: host)
       1   SETUP    P host_without_mlx_compat_loaded (fixtures used: host_with_saved_yum_state)
-      7   SETUP    P host_without_zfs (fixtures used: host)
+      3   SETUP    P host_without_zfs (fixtures used: host)
       1   SETUP    P host_with_perf (fixtures used: host_at_least_8_3, host_with_saved_yum_state)
       1   SETUP    P host_with_ring0_tests (fixtures used: host_with_saved_yum_state)
       1   SETUP    P host_with_runx_repo (fixtures used: host_with_saved_yum_state_toolstack_restart)
-     14   SETUP    P host_with_saved_yum_state (fixtures used: host)
-      4   SETUP    P host_with_saved_yum_state_toolstack_restart (fixtures used: host)
-     16   SETUP    P host_with_xfsprogs (fixtures used: _xfs_config, host, image_format)
+     10   SETUP    P host_with_saved_yum_state (fixtures used: host)
+      3   SETUP    P host_with_saved_yum_state_toolstack_restart (fixtures used: host)
+      2   SETUP    P host_with_xfsprogs (fixtures used: _xfs_config, host, image_format)
       1   SETUP    P host_with_zfs (fixtures used: host_with_saved_yum_state, host_without_zfs)
-     16   SETUP    P host_with_zfs (fixtures used: host_with_saved_yum_state, host_without_zfs, image_format)
-      3   SETUP    P host_with_zfsvol (fixtures used: host_with_saved_yum_state_toolstack_restart)
+      2   SETUP    P host_with_zfs (fixtures used: host_with_saved_yum_state, host_without_zfs, image_format)
+      2   SETUP    P host_with_zfsvol (fixtures used: host_with_saved_yum_state_toolstack_restart)
       2   SETUP    P largeblock_sr (fixtures used: host, image_format, unused_4k_disks)
-      1   SETUP    P _linstor_config
-      1   SETUP    P linstor_redundancy (fixtures used: pool_with_linstor)
-      1   SETUP    P linstor_sr (fixtures used: _linstor_config, linstor_redundancy, lvm_disks, pool_with_linstor, provisioning_type, storage_pool_name)
-      1   SETUP    P lvm_disks (fixtures used: pool_with_unused_512B_disk, provisioning_type, unused_512B_disks)
-      6   SETUP    P lvmohba_device_config
-     16   SETUP    P lvmohba_sr (fixtures used: host, image_format, lvmohba_device_config)
-      6   SETUP    P lvmoiscsi_device_config
-     16   SETUP    P lvmoiscsi_sr (fixtures used: host, image_format, lvmoiscsi_device_config)
-     16   SETUP    P lvm_sr (fixtures used: host, image_format, unused_512B_disks)
+      2   SETUP    P _linstor_config
+      2   SETUP    P linstor_redundancy (fixtures used: pool_with_linstor)
+      2   SETUP    P linstor_sr (fixtures used: _linstor_config, linstor_redundancy, lvm_disks, pool_with_linstor, provisioning_type, storage_pool_name)
+      2   SETUP    P lvm_disks (fixtures used: pool_with_unused_512B_disk, provisioning_type, unused_512B_disks)
+      2   SETUP    P lvmohba_device_config
+      2   SETUP    P lvmohba_sr (fixtures used: host, image_format, lvmohba_device_config)
+      2   SETUP    P lvmoiscsi_device_config
+      2   SETUP    P lvmoiscsi_sr (fixtures used: host, image_format, lvmoiscsi_device_config)
+      2   SETUP    P lvm_sr (fixtures used: host, image_format, unused_512B_disks)
       2   SETUP    P moosefs_device_config
       2   SETUP    P moosefs_sr (fixtures used: moosefs_device_config, pool_with_moosefs_enabled)
       3   SETUP    P nfs_device_config
       4   SETUP    P nfs_sr (fixtures used: host, image_format)
-      6   SETUP    P nfs_sr (fixtures used: host, image_format, nfs_device_config)
-      3   SETUP    P pool_with_ceph (fixtures used: pool_with_saved_yum_state, pool_without_ceph)
-      3   SETUP    P pool_with_glusterfs (fixtures used: _glusterfs_config, pool_with_saved_yum_state, pool_without_glusterfs)
-      1   SETUP    P pool_with_linstor (fixtures used: _linstor_config, hostA2, lvm_disks, pool_with_saved_yum_state)
+      4   SETUP    P nfs_sr (fixtures used: host, image_format, nfs_device_config)
+      2   SETUP    P pool_with_ceph (fixtures used: pool_with_saved_yum_state, pool_without_ceph)
+      2   SETUP    P pool_with_glusterfs (fixtures used: _glusterfs_config, pool_with_saved_yum_state, pool_without_glusterfs)
+      2   SETUP    P pool_with_linstor (fixtures used: _linstor_config, hostA2, lvm_disks, pool_with_saved_yum_state)
       2   SETUP    P pool_with_moosefs_enabled (fixtures used: pool_with_moosefs_installed)
       2   SETUP    P pool_with_moosefs_installed (fixtures used: pool_with_saved_yum_state)
-      3   SETUP    P pool_without_ceph (fixtures used: host)
-      3   SETUP    P pool_without_glusterfs (fixtures used: host)
-      9   SETUP    P pool_with_saved_yum_state (fixtures used: host)
-     10   SETUP    P sr_disk_wiped (fixtures used: host, unused_512B_disks)
-      1   SETUP    P storage_pool_name (fixtures used: provisioning_type)
-      6   SETUP    P _xfs_config
-      3   SETUP    P _xfs_config_on_hostA2
-      3   SETUP    P _xfs_config_on_hostB1
-     16   SETUP    P xfs_sr (fixtures used: _xfs_config, host_with_xfsprogs, image_format, unused_512B_disks)
-     21   SETUP    P xfs_sr_on_hostA2 (fixtures used: _xfs_config_on_hostA2, hostA2_with_xfsprogs, image_format, unused_512B_disks)
-     22   SETUP    P xfs_sr_on_hostB1 (fixtures used: _xfs_config_on_hostB1, hostB1_with_xfsprogs, image_format, unused_512B_disks)
+      2   SETUP    P pool_without_ceph (fixtures used: host)
+      2   SETUP    P pool_without_glusterfs (fixtures used: host)
+      8   SETUP    P pool_with_saved_yum_state (fixtures used: host)
+      5   SETUP    P sr_disk_wiped (fixtures used: host, unused_512B_disks)
+      2   SETUP    P storage_pool_name (fixtures used: provisioning_type)
+      2   SETUP    P _xfs_config
+      4   SETUP    P _xfs_config_on_hostA2
+      4   SETUP    P _xfs_config_on_hostB1
+      2   SETUP    P xfs_sr (fixtures used: _xfs_config, host_with_xfsprogs, image_format, unused_512B_disks)
+      4   SETUP    P xfs_sr_on_hostA2 (fixtures used: _xfs_config_on_hostA2, hostA2_with_xfsprogs, image_format, unused_512B_disks)
+      4   SETUP    P xfs_sr_on_hostB1 (fixtures used: _xfs_config_on_hostB1, hostB1_with_xfsprogs, image_format, unused_512B_disks)
       1   SETUP    P xtf_runner (fixtures used: host_with_git_and_gcc_and_py3)
-     16   SETUP    P zfs_sr (fixtures used: host, image_format, zpool_vol0)
-      3   SETUP    P zfsvol_sr (fixtures used: host, host_with_zfsvol, sr_disk_wiped)
-     17   SETUP    P zpool_vol0 (fixtures used: host_with_zfs, sr_disk_wiped)
+      2   SETUP    P zfs_sr (fixtures used: host, image_format, zpool_vol0)
+      2   SETUP    P zfsvol_sr (fixtures used: host, host_with_zfsvol, sr_disk_wiped)
+      3   SETUP    P zpool_vol0 (fixtures used: host_with_zfs, sr_disk_wiped)
       1 SETUP    S disks (fixtures used: pools_hosts_by_name_or_ip, pytestconfig)
       1 SETUP    S enabled_pci_uuid (fixtures used: enabled_pgpu_uuid, host)
       1 SETUP    S enabled_pgpu_uuid (fixtures used: host)
@@ -193,8 +193,8 @@
       1 SETUP    S hosts (fixtures used: pytestconfig)
       1 SETUP    S hosts_with_xo (fixtures used: hosts, registered_xo_cli)
       1 SETUP    S host_with_hsts (fixtures used: host)
-     44 SETUP    S image_format['qcow2']
-     44 SETUP    S image_format['vhd']
+      1 SETUP    S image_format['qcow2']
+      1 SETUP    S image_format['vhd']
       1 SETUP    S local_sr_on_hostA1 (fixtures used: hostA1)
       1 SETUP    S local_sr_on_hostA2 (fixtures used: hostA2)
       1 SETUP    S pools_hosts_by_name_or_ip (fixtures used: hosts)
@@ -292,45 +292,45 @@
       1     TEARDOWN M unix_vm_with_vtpm
       2     TEARDOWN M unsealed_windows_vm_and_snapshot
       1     TEARDOWN M vdi_on_cephfs_sr
-      6     TEARDOWN M vdi_on_ext_sr
+      4     TEARDOWN M vdi_on_ext_sr
       1     TEARDOWN M vdi_on_glusterfs_sr
       2     TEARDOWN M vdi_on_largeblock_sr
       1     TEARDOWN M vdi_on_linstor_sr
       1     TEARDOWN M vdi_on_moosefs_sr
       2     TEARDOWN M vdi_on_nfs_sr
-      7     TEARDOWN M vdi_on_xfs_sr
-      7     TEARDOWN M vdi_on_zfs_sr
+      4     TEARDOWN M vdi_on_xfs_sr
+      4     TEARDOWN M vdi_on_zfs_sr
       1     TEARDOWN M vdi_on_zfsvol_sr
       1     TEARDOWN M vdis
       6     TEARDOWN M vm_on_cephfs_sr
-     12     TEARDOWN M vm_on_ext_sr
+      8     TEARDOWN M vm_on_ext_sr
       6     TEARDOWN M vm_on_glusterfs_sr
       6     TEARDOWN M vm_on_largeblock_sr
-      5     TEARDOWN M vm_on_linstor_sr
-     12     TEARDOWN M vm_on_lvmohba_sr
-     12     TEARDOWN M vm_on_lvmoiscsi_sr
-     12     TEARDOWN M vm_on_lvm_sr
-      4     TEARDOWN M vm_on_moosefs_sr
+      6     TEARDOWN M vm_on_linstor_sr
+      8     TEARDOWN M vm_on_lvmohba_sr
+      8     TEARDOWN M vm_on_lvmoiscsi_sr
+      8     TEARDOWN M vm_on_lvm_sr
+      6     TEARDOWN M vm_on_moosefs_sr
       2     TEARDOWN M vm_on_nfs_sr
-     12     TEARDOWN M vm_on_xfs_sr
-     12     TEARDOWN M vm_on_zfs_sr
-      3     TEARDOWN M vm_on_zfsvol_sr
-    118     TEARDOWN M vm_ref[None]
+      8     TEARDOWN M vm_on_xfs_sr
+      8     TEARDOWN M vm_on_zfs_sr
+      5     TEARDOWN M vm_on_zfsvol_sr
+    126     TEARDOWN M vm_ref[None]
       1     TEARDOWN M vm_with_vbds
       4     TEARDOWN M windows_vm
-      3   TEARDOWN P cephfs_device_config
-      3   TEARDOWN P cephfs_sr
-      9   TEARDOWN P ext_sr
+      2   TEARDOWN P cephfs_device_config
+      2   TEARDOWN P cephfs_sr
+      2   TEARDOWN P ext_sr
       3   TEARDOWN P formatted_and_mounted_ext4_disk
       1   TEARDOWN P fsp_config
       1   TEARDOWN P fsp_sr
-      3   TEARDOWN P gluster_disk
-      3   TEARDOWN P _glusterfs_config
-      3   TEARDOWN P glusterfs_device_config
-      3   TEARDOWN P glusterfs_sr
-      3   TEARDOWN P gluster_volume_started
-     21   TEARDOWN P hostA2_with_xfsprogs
-     22   TEARDOWN P hostB1_with_xfsprogs
+      2   TEARDOWN P gluster_disk
+      2   TEARDOWN P _glusterfs_config
+      2   TEARDOWN P glusterfs_device_config
+      2   TEARDOWN P glusterfs_sr
+      2   TEARDOWN P gluster_volume_started
+      4   TEARDOWN P hostA2_with_xfsprogs
+      4   TEARDOWN P hostB1_with_xfsprogs
       1   TEARDOWN P host_no_sdn_controller
       1   TEARDOWN P host_with_dom0_tests
       1   TEARDOWN P host_with_dynamically_disabled_ept_sp
@@ -340,49 +340,49 @@
       1   TEARDOWN P host_with_netdata
       1   TEARDOWN P host_without_mlx_card
       1   TEARDOWN P host_without_mlx_compat_loaded
-      7   TEARDOWN P host_without_zfs
+      3   TEARDOWN P host_without_zfs
       1   TEARDOWN P host_with_perf
       1   TEARDOWN P host_with_ring0_tests
       1   TEARDOWN P host_with_runx_repo
-     14   TEARDOWN P host_with_saved_yum_state
-      4   TEARDOWN P host_with_saved_yum_state_toolstack_restart
-     16   TEARDOWN P host_with_xfsprogs
-     17   TEARDOWN P host_with_zfs
-      3   TEARDOWN P host_with_zfsvol
+     10   TEARDOWN P host_with_saved_yum_state
+      3   TEARDOWN P host_with_saved_yum_state_toolstack_restart
+      2   TEARDOWN P host_with_xfsprogs
+      3   TEARDOWN P host_with_zfs
+      2   TEARDOWN P host_with_zfsvol
       2   TEARDOWN P largeblock_sr
-      1   TEARDOWN P _linstor_config
-      1   TEARDOWN P linstor_redundancy
-      1   TEARDOWN P linstor_sr
-      1   TEARDOWN P lvm_disks
-      6   TEARDOWN P lvmohba_device_config
-     16   TEARDOWN P lvmohba_sr
-      6   TEARDOWN P lvmoiscsi_device_config
-     16   TEARDOWN P lvmoiscsi_sr
-     16   TEARDOWN P lvm_sr
+      2   TEARDOWN P _linstor_config
+      2   TEARDOWN P linstor_redundancy
+      2   TEARDOWN P linstor_sr
+      2   TEARDOWN P lvm_disks
+      2   TEARDOWN P lvmohba_device_config
+      2   TEARDOWN P lvmohba_sr
+      2   TEARDOWN P lvmoiscsi_device_config
+      2   TEARDOWN P lvmoiscsi_sr
+      2   TEARDOWN P lvm_sr
       2   TEARDOWN P moosefs_device_config
       2   TEARDOWN P moosefs_sr
       3   TEARDOWN P nfs_device_config
-     10   TEARDOWN P nfs_sr
-      3   TEARDOWN P pool_with_ceph
-      3   TEARDOWN P pool_with_glusterfs
-      1   TEARDOWN P pool_with_linstor
+      8   TEARDOWN P nfs_sr
+      2   TEARDOWN P pool_with_ceph
+      2   TEARDOWN P pool_with_glusterfs
+      2   TEARDOWN P pool_with_linstor
       2   TEARDOWN P pool_with_moosefs_enabled
       2   TEARDOWN P pool_with_moosefs_installed
-      3   TEARDOWN P pool_without_ceph
-      3   TEARDOWN P pool_without_glusterfs
-      9   TEARDOWN P pool_with_saved_yum_state
-     10   TEARDOWN P sr_disk_wiped
-      1   TEARDOWN P storage_pool_name
-      6   TEARDOWN P _xfs_config
-      3   TEARDOWN P _xfs_config_on_hostA2
-      3   TEARDOWN P _xfs_config_on_hostB1
-     16   TEARDOWN P xfs_sr
-     21   TEARDOWN P xfs_sr_on_hostA2
-     22   TEARDOWN P xfs_sr_on_hostB1
+      2   TEARDOWN P pool_without_ceph
+      2   TEARDOWN P pool_without_glusterfs
+      8   TEARDOWN P pool_with_saved_yum_state
+      5   TEARDOWN P sr_disk_wiped
+      2   TEARDOWN P storage_pool_name
+      2   TEARDOWN P _xfs_config
+      4   TEARDOWN P _xfs_config_on_hostA2
+      4   TEARDOWN P _xfs_config_on_hostB1
+      2   TEARDOWN P xfs_sr
+      4   TEARDOWN P xfs_sr_on_hostA2
+      4   TEARDOWN P xfs_sr_on_hostB1
       1   TEARDOWN P xtf_runner
-     16   TEARDOWN P zfs_sr
-      3   TEARDOWN P zfsvol_sr
-     17   TEARDOWN P zpool_vol0
+      2   TEARDOWN P zfs_sr
+      2   TEARDOWN P zfsvol_sr
+      3   TEARDOWN P zpool_vol0
       1 TEARDOWN S disks
       1 TEARDOWN S enabled_pci_uuid
       1 TEARDOWN S enabled_pgpu_uuid
@@ -398,8 +398,8 @@
       1 TEARDOWN S hosts
       1 TEARDOWN S hosts_with_xo
       1 TEARDOWN S host_with_hsts
-     44 TEARDOWN S image_format['qcow2']
-     44 TEARDOWN S image_format['vhd']
+      1 TEARDOWN S image_format['qcow2']
+      1 TEARDOWN S image_format['vhd']
       1 TEARDOWN S local_sr_on_hostA1
       1 TEARDOWN S local_sr_on_hostA2
       1 TEARDOWN S pools_hosts_by_name_or_ip

Are you sure you're comparing the right versions?

vxgmichel · 2026-05-21T16:46:46Z

I generated a quick report from those diff, the difference is quite impressive:

More often with per-axis test ordering:
Total diff: +21
[ +8] (128 -> 120) M vm_ref[None]
[ +2] (  6 ->   4) M vm_on_moosefs_sr
[ +2] (  5 ->   3) M vm_on_zfsvol_sr
[ +1] (  6 ->   5) M vm_on_linstor_sr
[ +1] (  4 ->   3) P _xfs_config_on_hostA2
[ +1] (  4 ->   3) P _xfs_config_on_hostB1
[ +1] (  2 ->   1) P _linstor_config
[ +1] (  2 ->   1) P linstor_redundancy
[ +1] (  2 ->   1) P linstor_sr
[ +1] (  2 ->   1) P lvm_disks
[ +1] (  2 ->   1) P pool_with_linstor
[ +1] (  2 ->   1) P storage_pool_name

Less often with per-axis test ordering:
Total diff: -349
[-43] (  1 ->  44) S image_format['qcow2']
[-43] (  1 ->  44) S image_format['vhd']
[-18] (  4 ->  22) P hostB1_with_xfsprogs
[-18] (  4 ->  22) P xfs_sr_on_hostB1
[-17] (  4 ->  21) P hostA2_with_xfsprogs
[-17] (  4 ->  21) P xfs_sr_on_hostA2
[-14] (  3 ->  17) P zpool_vol0
[-14] (  2 ->  16) P host_with_xfsprogs
[-14] (  2 ->  16) P host_with_zfs
[-14] (  2 ->  16) P lvm_sr
[-14] (  2 ->  16) P lvmohba_sr
[-14] (  2 ->  16) P lvmoiscsi_sr
[-14] (  2 ->  16) P xfs_sr
[-14] (  2 ->  16) P zfs_sr
[ -7] (  2 ->   9) P ext_sr
[ -5] (  5 ->  10) P sr_disk_wiped
[ -4] ( 10 ->  14) P host_with_saved_yum_state
[ -4] (  8 ->  12) M vm_on_ext_sr
[ -4] (  8 ->  12) M vm_on_lvm_sr
[ -4] (  8 ->  12) M vm_on_lvmohba_sr
[ -4] (  8 ->  12) M vm_on_lvmoiscsi_sr
[ -4] (  8 ->  12) M vm_on_xfs_sr
[ -4] (  8 ->  12) M vm_on_zfs_sr
[ -4] (  3 ->   7) P host_without_zfs
[ -4] (  2 ->   6) P _xfs_config
[ -4] (  2 ->   6) P lvmohba_device_config
[ -4] (  2 ->   6) P lvmoiscsi_device_config
[ -3] (  4 ->   7) M vdi_on_xfs_sr
[ -3] (  4 ->   7) M vdi_on_zfs_sr
[ -2] (  4 ->   6) M vdi_on_ext_sr
[ -2] (  4 ->   6) P nfs_sr
[ -1] (  8 ->   9) P pool_with_saved_yum_state
[ -1] (  3 ->   4) P host_with_saved_yum_state_toolstack_restart
[ -1] (  2 ->   3) P _glusterfs_config
[ -1] (  2 ->   3) P cephfs_device_config
[ -1] (  2 ->   3) P cephfs_sr
[ -1] (  2 ->   3) P gluster_disk
[ -1] (  2 ->   3) P gluster_volume_started
[ -1] (  2 ->   3) P glusterfs_device_config
[ -1] (  2 ->   3) P glusterfs_sr
[ -1] (  2 ->   3) P host_with_zfsvol
[ -1] (  2 ->   3) P pool_with_ceph
[ -1] (  2 ->   3) P pool_with_glusterfs
[ -1] (  2 ->   3) P pool_without_ceph
[ -1] (  2 ->   3) P pool_without_glusterfs
[ -1] (  2 ->   3) P zfsvol_sr

glehmann · 2026-05-22T17:36:36Z

I started a full run with vhd and qcow2. Wait & see

stormi · 2026-06-10T15:31:00Z

Rebased on master as it was starting to lag behind. I'll apply Gaëtan's suggestions now.

pytest's default ordering could interleave image_format=vhd/qcow2 tests in a way that repeatedly destroys and recreates SRs. It could also choose to cross package boundaries as an attempt to optimize test runs, but we'd prefer to see tests logically grouped, for easier log analysis and fixture management (which is perfectible and which we'll have to improve anyway). We start with pytest's computed ordering, then regroup tests by parametrized axis (currently only image_format) and by leaf python package, in order to keep related tests together and reduce needless costly context switching. Axis-less tests intentionally share the same ordering level as the first parametrized axis so they stay near their original position instead of being grouped at the beginning of the run. Interfering with pytest's ordering is not something to do lightly, so it's possible that we may discover issues caused by this. But meanwhile, this has shown a notable reduction (10%) of the number of setup/teardown operations when both vhd and qcow2 image formats are set, and test order closer to what we'd intuitively expect. Signed-off-by: Samuel Verschelde <stormi-xcp@ylix.fr>

stormi · 2026-06-10T16:14:03Z

This PR is ready. However I haven't tested the last pushes (rebase + apply Gaëtan's suggestions) thoroughly. I only ran a --setup-plan on tests/ in order to detect obvious failures. @vxgmichel could you run the test you had done previously in order to verify the output?

vxgmichel · 2026-06-16T16:19:57Z

@stormi

Could you run the test you had done previously in order to verify the output?

Just to be clear, it wasn't a test, just a script to summarize the differences of the two --setup-plan (with and without per-axis ordering). I just updated my previous comment with the new output. You can check the last edit and see that even though the number of tests changed, the script still reports +21/-349 fixture runs.

stormi · 2026-06-16T16:21:24Z

@stormi

Could you run the test you had done previously in order to verify the output?

Just to be clear, it wasn't a test, just a script to summarize the differences of the two --setup-plan (with and without per-axis ordering). I just updated my previous comment with the new output. You can check the last edit and see that even though the number of tests changed, the script still reports +21/-349 fixture runs.

Thanks, that's the "test" I had in mind. It tends to show that I haven't broken the PR when I applied the review suggestions. Then I think we're good to go. Ping @rzr @ydirson.

stormi · 2026-06-17T13:25:05Z

I think we'll give this PR one more day for additional reviews, then merge if none.

stormi · 2026-06-22T21:48:40Z

I think we'll give this PR one more day for additional reviews, then merge if none.

As promised/warned, merging!

stormi requested review from glehmann and rzr May 20, 2026 16:27

stormi requested a review from a team as a code owner May 20, 2026 16:27

stormi commented May 20, 2026

View reviewed changes

Comment thread conftest.py Outdated

stormi force-pushed the stormi/ordering_constraints branch from d945be8 to 9568250 Compare May 20, 2026 16:45

ydirson requested changes May 21, 2026

View reviewed changes

glehmann approved these changes May 21, 2026

View reviewed changes

glehmann mentioned this pull request Jun 2, 2026

Fix the package scope of xfs_sr_on_hostA2 and xfs_sr_on_hostB1 #547

Merged

stormi requested a review from ydirson June 10, 2026 15:28

stormi force-pushed the stormi/ordering_constraints branch from 9568250 to 7ddd7a2 Compare June 10, 2026 15:30

stormi force-pushed the stormi/ordering_constraints branch from 7ddd7a2 to c6afd80 Compare June 10, 2026 15:47

stormi force-pushed the stormi/ordering_constraints branch from c6afd80 to 4fe1e21 Compare June 10, 2026 16:05

stormi merged commit 9bbeb54 into master Jun 22, 2026
15 of 17 checks passed

stormi deleted the stormi/ordering_constraints branch June 22, 2026 21:48

Uh oh!

Conversation

stormi commented May 20, 2026

Uh oh!

Uh oh!

ydirson left a comment

Choose a reason for hiding this comment

Uh oh!

glehmann May 21, 2026

Choose a reason for hiding this comment

Uh oh!

stormi May 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

glehmann May 21, 2026

Choose a reason for hiding this comment

Uh oh!

glehmann commented May 21, 2026

Uh oh!

vxgmichel commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

glehmann commented May 22, 2026

Uh oh!

stormi commented Jun 10, 2026

Uh oh!

stormi commented Jun 10, 2026

Uh oh!

vxgmichel commented Jun 16, 2026

Uh oh!

stormi commented Jun 16, 2026

Uh oh!

stormi commented Jun 17, 2026

Uh oh!

stormi commented Jun 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vxgmichel commented May 21, 2026 •

edited

Loading